1 


Wearable-In-The-Loop: Building Wearables for 

Industry 4.0 

Saul Delabrida, Thiago D’Angelo, Theo Lins, Ricardo A. R. Oliveira, Mark Billinghurst, Ross T. Smith, 

Bruce H. Thomas and Antonio Loureiro 


Abstract — Cyber-Physical Systems (CPS) is a new approach for 
developing smart factories following the Industry 4.0 concept. 
Although these systems tend to be self-adapting, humans also 
play an important role in the Industry 4.0 context. Humans 
aspects need to be taken into consideration when designing 
these systems, and People-Centric applications are important 
for this purpose. This paper discusses the use of wearable 
computing for building People-Centric applications in Industry 
4.0 smart factories, especially focusing on the human aspects. A 
methodology named wearable-in-the-loop is presented to improve 
the design of human interfaces within Industry 4.0 applications 
and CPS. The paper presents a wearable prototype and its 
evaluation. This work represents the first step of the wearable- 
in-the-loop. 

Index Terms — Human-in-the-loop, Wearable, Wearable-in-the- 
loop, Industry 4.0, Cyber-Physical System. 

I. Introduction 

T HE new age of industry has been named Industry 4.0 due 
to the relation with the coming of the fourth industrial 
revolution. A major technology of Industry 4.0 are Cyber- 
Physical Systems (CPS) that are systems composed of sensors, 
actuators, network and computer algorithms that work together 
to control physical entities. The Industry 4.0 uses the Internet 
of Things (IoT) as the main communication infra-structure 
between the physical entities [1], 

An important related research area is wearable comput- 
ing [2], which aims to provide constant access to computing 
and communications to people using wearable technology. 
Today there are a wide range of different wearable computing 
systems, including those related to applications providing 
health information, supporting sports, providing security and 
other applications. Wearable computing systems are strongly 
related to the Industry 4.0 objectives, and can provide an 
Human Machine Interface (HMI) for Industry 4.0 applications. 
Previous research has shown how smartphones, tablets and 
other mobile user interfaces can be used to support interaction 
between users and smart components in a factory [3]. 

In order to design interfaces for Industry 4.0 applica- 
tions it is important to understand human behavior in an 
industrial setting. One way to understand human behavior is 
through social sensing and using virtual sensors. Although 
these sources are good for sensing user behavior, they may 
be not good alternatives for industrial scenarios, which may 
not have worker behavior data sensing. Furthermore, Nunes 
et.al [4] show that the current state-of-the-art for including 
human-behavior models in CPS are very specific for each 
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application and need to be incorporated into a feedback control 
methodology. 

This paper introduces the Wearable-In-The-Loop (WITL) 
concept for improving user experience in Industry 4.0 applica- 
tions and presents an architecture model for building wearable 
appliances. 

Our contributions are: 

• A methodology for improving human interaction experi- 
ences in a factory, 

• A methodology for building wearable appliances, 

• A discussion about human and technological aspects of 
using wearables for human-computer interaction, and 

• A classification of different types of human feedback for 
industry 4.0 applications providing wearable examples for 
each one. 

The main contribution is the formal methodology for including 
the human aspects in the industrial environment, as well as, the 
inclusion of a methodology that makes it possible to provide a 
new mature stage of the wearable as a worker’s tool, Wearable - 
in-the-Loop methodology. We consider the use of wearables 
for sensing data as input for an architecture which is able 
to execute algorithms and support the specialist evaluation. 
The system architecture provides a set of services in order 
to understand the human behavior in an Industry 4.0 facility. 
These kinds of services are an announced challenge [4] of the 
community for the inclusion of the human in the CPS. We 
formalize a methodology for building wearable appliances. 

The rest of this paper is organized as follows: Section II 
presents Industry 4.0 concepts and describes CPS modules. 
Section III presents wearable computer concepts and a brief 
discussion about wearable accessories and wearable appli- 
ances. Section IV discusses human-in-the-loop aspects. Sec- 
tion V discusses about wearable use in Industry 4.0. Section V 
presents the wearable-in-the-loop and a methodology for build- 
ing wearable appliances and shows a example of how to build a 
wearable system using this methodology. Finally, conclusions 
and directions for future work are presented. 

II. Industry 4.0 - Towards the Smart Factory 

Industry 4.0 emerged in 201 1 and represents the latest new 
era of industry development. The advance of IoT contributed to 
the consolidation of Industry 4.0. The main difference between 
Industry 4.0 and Industry 3.0 is the focus on Cyber-Physical 
systems [5], which supports communication among industry 
entities such as real devices with information processing and 
virtual objects. In contrast. Industry 3.0 systems handle only 
the automation process. 
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The challenge of Industry 4.0 is to build smart factories 
that integrate robots, control systems, instruments, monitor- 
ing tools, sensors and use Human Machine Interface (HMI) 
systems to keep the humans in the loop. The most common 
interface is the one that users hold with their hands to 
operate [3], [6], such as smartphones and tablets. 

A. Robots 

The IoT is used in Industry 4.0 to support the intercon- 
nection of various devices. This allows robots, which are part 
of the production process, to have their actions coordinated 
by humans using information from the production database. 
Thus they can complete the tasks intelligently, helping humans 
who do not have to worry about small logistical details. 
For an effective, successful, and safe support of humans in 
physical tasks, it is necessary that robots interact smoothly 
and intuitively with their human counterparts and that humans 
are properly trained for this kind of human-machine collabo- 
ration [1], 

B. Control Systems 

Production control systems are responsible for joining the 
CPS with the human interface and other industrial technologies 
through standardized interfaces. Thus all data generated by the 
production process is made available to users of the industry. 
The increased information collected and data handling per- 
formed by the control system brings more flexibility to the 
production process [5], 

C. Monitoring Tool 

The industry monitoring process is essential for an effective 
operation of the whole system, aiming at an improvement 
in quality. The production components require adjustments 
during the production process, due to mass customization and 
greater flexibility in production, and in order to keep the pro- 
duction pipeline operating with a high yield and quality. Thus 
monitoring tools enable users to monitor the entire system and 
perform timely adjustments, avoiding manual interventions 
that are usually performed with a lack of information. 

D. Human Machine Interface 

The efficiency of the production process depends on the 
quality of the HMI, which is the interface between the human 
and the CPS. One way for doing this is through using 
Augmented Reality, technology that allows virtual imagery to 
be overlaid on the real world. Paelke [7] shows an augmented 
reality system that supports human workers in a rapidly 
changing production environment. 

III. Wearable Computing 

Wearable computing is another emergent research area 
which uses embedded smart computers on human body to 
collect information about the user or about the context around 
the users. Smartwatches and the HoloLens Augmented Reality 


(AR) head-mounted display (HMD) 1 are two examples of 
wearable computing systems currently available. Users can 
change their behavior during daily tasks based on feedback 
received by wearable devices [8]. 

In general, we can classify any wearable device into one of 
these following two categories: (1) wearable appliances and 
(2) wearable accessories [9]. The wearable appliance category 
consists of wearable devices which can perform their task 
without the aid of another device. Meanwhile, the wearable 
accessory category consists of wearable devices which need 
to communicate with another device to fulfill their purpose. 

Generally, wearable appliances demand more complex hard- 
ware and software features in order to process and deliver 
to the user all necessary information related to their tasks. 
Therefore, a wearable appliance demands a smart sensor 
architecture level, taking advantage of the abstraction level 
provided by the operating system in order to fulfill its purpose 
without the aid of another device. 

The use of wearable computing systems integrated with 
Industry 4.0 is being explored. For example, [10] uses smart- 
glasses as a tool to give support in maintenance activities. 
Wearable computing offers hands-free interaction capabilities 
and in-situ information presentation on the factory floor. The 
wearable computing system provides a unique platform to host 
a range of sensor technologies. Sensors can monitor the user’s 
health, emotions, and current task. 

IV. Human-in-the-Loop 

In the context of this paper, Human-in-the-loop (HITL) rep- 
resents the interaction of the human into the industrial process. 
Cyber-Physical systems demand a high level of integration 
with HITL applications. Three approaches are considered to 
build HITL applications. 1) Human control the system. 2) 
System monitoring humans. 3) A hybrid combination of both. 

Nunes et.al [4] present challenges in respect to this inte- 
gration. The human physical/physiological state can influence 
the cyber-physical systems behavior. For example, let us 
consider teleoperation activities where the user is assisted by 
sensors distributed in the industrial environment. The current 
human condition while teleoperating the device can influence 
the performance result and his/her productivity. Furthermore, 
Nunes et. al indicated that the current solutions for integrating 
human conditions and CPS, which provide a formal model 
to give feedback on humans behavior, can be too generic 
or specific for each application. The wearable can perform 
a relevant role in this process by capturing the data from user, 
but it is necessary to consider the current state of technology 
in order to integrate output from wearable devices with other 
modules. The combination of smart factory components, the 
data collect from sensors and machine learning techniques, 
provide the chance to add blocks to the architecture in from 
which frequent feedback can be collected and user analysis 
can be estimated. 

User experience in an unfriendly user environment is a 
typical scenario in the industrial context. In this scenario, the 
use of sensors for monitoring the user are cmcial for health 

1 https://www.microsoft.com/microsoft-holoIens/en-us 
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and safety. Cameras are examples of sensors used for this 
action but this usually depends on human analysis to identify 
critical events. The integration of wearables with other sensors 
can create more efficient monitoring. For example, the use of 
accelerometers on the user’s body can identify a person’s fall 
and emit alerts for fast assistance. 

V. Wearable and Industry 4.0 

The use of wearable technology goes beyond the use of 
an accessory or a peripheral hardware to smartphones on 
industrial applications. In the Industry 4.0, wearable computers 
work as a tool instead of an accessory. Wearable computers 
can be used as an appliance to provide support for the 
development of industrial tasks that depend on humans, or for 
manipulating equipment or monitoring them. To provide this 
feature, a set of changes to the typical wearable architectures 
are necessary. For example, an integration of data collected 
from the wearable computer and Cyber-Physical systems could 
provide a better perception of how to develop a particular 
task. In the next moment, the wearable technology can be 
adapted and receive new functionalities based on the results 
and obtained knowledge after the task be evaluated by the 
specialists. This section discusses human aspects and gives 
examples of how wearable technology can become a valuable 
tool for humans perform their tasks in an Industry 4.0 setting. 
It presents a set of concepts and requirements that should be 
considered for building Wearable -in-the-loop applications. 

A. Human Aspects for Industry 4.0 

In an Industry 4.0 environment the human is considered as 
a vital part of the system, especially due to the fact that some 
industrial tasks rely on human cooperation. In this context, 
we define Industrial Cyber-physical systems as distributed 
systems that contain humans as entities. In this case, it is 
necessary to understand the human aspects that effect the 
human performance. One of the challenges of Industry 4.0 
applications is the need to provide available technology to 
monitor these aspects. [11] presents the unique aspects that 
characterize how humans perform their tasks. These aspects 
are: 

• Cognitive system, 

• Motivation, 

• Preferences, 

• Social behavior, 

• Emotion, 

• Intraindividual variability and change, and 

• Individual differences. 

Although the study of these aspects is mainly explored 
in psychology, advances in technology may provide mecha- 
nisms to support further investigations. For example, wearable 
computing can capture sensing data, and machine learning 
and data mining techniques can be used for assessment by 
a psychology professional. Moreover, in some Industry 4.0 
situations, it is necessary to enhance the health aspects of 
the humans performing their tasks. For this purpose, wearable 
technologies are ideal, since most of them are designed to 
monitor body conditions for health and sports purposes. 


This demand for technologies to support human assessment 
also raises discussions about privacy. New ethical questions 
emerge in this scenario and using unnamed data in studies 
may not be more efficient, once the individual may be the 
object of study. 

Wearable computers are tools able to collect and provide 
information for workers while they are performing their tasks 
on the industrial environment. Some human aspects can be 
influenced by this interaction, such as emotion, preferences 
and motivation. 

The next section discusses the mechanism for collecting 
data from a user and providing them with information about 
the context. 

B. Human as the Sensor 

The challenge of cyber-physical systems and human-in-the- 
loop integration is to relate humans aspects such as physiologi- 
cal and physical elements. Both can influence the development 
of tasks. Wearable technologies can collect enough data to 
evaluate the human state while he or she is working. The 
challenge is to build technologies that analyze data collected 
from the user and received from the cyber-physical systems 
modules in order to identify the influence of the human on the 
system in real time. 

C. Providing User Feedback 

In some cases, the human interventions with the CPS 
systems are necessary. In this case, the human needs to receive 
information from the system to make decisions or provide data 
for the CPS components operation. An example of this is using 
a wearable computer for the emission of alerts to the user in 
case of risk in the industrial environment. 

There are different ways to give feedback to the user. Visual, 
auditory and haptic feedback are some examples of how a user 
can be alerted to some context condition. We classify the types 
of feedback in industrial scenarios into three categories: 1) 
Frequent Feedback; 2) Non-Frequent Feedback and 3) Critical 
Feedback. This section describes each category and presents 
some examples of wearable computers that can interact with 
humans in this way and relate them to situations where each 
one is appropriate. 

Frequent Feedback: The frequent feedback category means 
that continuous information is necessary to perform human ac- 
tions. Head Mounted Displays (HMD) are the most appropriate 
wearable technology for visual and frequent feedback. There 
are two types of HMDs: 1) A optical see-through HMD uses 
a transparent lens to show the virtual information associated 
with the real view; 2) A video see-through HMD shows the 
live video view captured by a camera overlaid with virtual 
information. 

Non-Frequent Feedback: The non-frequent feedback cate- 
gory represents occasional feedback that does not influence the 
user’s action. A Smartwatch an example of a wearable device 
providing non-frequent user feedback. These devices have a 
design similar to wristwatches, which are devices commonly 
used by most people, and so they are the most conventional 
devices for wearable end users. Furthermore, most of these 
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TABLE I 

Human Feedback Category 


Category 

Wearable 

Example 

Type of 

Application 

Frequent Feedback 

HMD 

Overlapping information 

Non-Frequent Feedback 

Smart watch 

Text messages or vibration 

Critical Feedback 

Haptic Glove 

Teleoperation, Training 


wearable computers are equipped with operating systems that 
support mature APIs for application development. This fact 
increases the interest of industry in developing specific appli- 
cations to support industrial processes. The disadvantage of 
this equipment is their limited screen size and low resolution. 
Usually, these devices provide text information to the user or 
simple images with a low quality of information. On the other 
hand, they can use different vibration patterns to allow the 
user to recognize the type of alert emitted. 

Critical Feedback: Critical feedback is necessary for situ- 
ations where the user is at risk. Haptic gloves are wearable 
devices that can simulate tactic sensing to provide feedback 
to the user in virtual reality environments. The device contains 
sensors to collect data from user movement, and can be 
used for training purposes in industry 4.0 applications, or 
for teleoperation. Haptic gloves can also avoid undesired user 
behavior according based on CPS feedback. This variety of 
system expands its utility when it is integrated with other 
techniques, such as an HMD with virtual reality immersion. 
Various operations in the industrial environment require access 
to non user-friendly areas. This fact implies risks to human 
health and even risk of death, depending on how critical the 
area is. Companies, look for alternatives that reduce these 
risks and allow the operator to work safely. In this scenario, 
the integration of user’s wearing haptic gloves working syn- 
chronously with robots in an Industry 4.0 setting is imminent. 
Furthermore, haptic gloves can aid user operations to reduce 
the muscles fatigue or to give support to repetitive human 
tasks [12], 

All of these wearable technologies have specific applications 
in industrial scenarios. Table I summarizes the features and 
examples of each category presented. 

VI. Wearables-in-the-loop for Industry 4.0 

Considering wearable systems as a worker tool, the WITL 
provides a methodology for getting a new maturation stage of 
the wearable systems, through the feedback and the resulting 
analysis after specialist evaluation. This section describes an 
methodology for continuous learning and improvements for 
HITL applications and an approach for building a wearable 
appliance. 

A. Wearable-in-the-Loop 

The Wearable-in-the-Loop is a methodology proposed for 
designing and evaluating a wearable system in an industrial 
scenario. The methodology is composed by the integration of 
modules that work together to provide new knowledge about 
the HITL application. Figure 1 shows the integration among 
the modules. The methodology is composed of three stages; 


(1) Sensing, (2) Analysis, and (3) Maturity. Next, each one is 
described in turn. 

a) Sensing Stage: In this stage data collected from CPS 
modules and workers is stored in the Raw Database. There are 
several types of data that can be collected from the Workers 
as described in the Section V-B. 

b) Analysis Stage: In this stage there are two main 
activities. 1) Real-time data provided by CPS reports and Hu- 
man reports. This data represents real-time information about 
CPS components. The system can also provide information 
to support the worker’s task operation. 2) A Data Analysis 
System is an infrastructure to execute advanced algorithms in 
order to discover new features of an WITL application and 
learn how to improve the user experience. Both types of data 
analysis may be used to generate feedback for the operational 
activities and strategic decisions of the worker. 

c) Achieving Mature Stage: The results of Data Analy- 
sis System are evaluated by specialists (researchers, doctors, 
psychologist and others) in other to get new knowledge. The 
results are helpful to build or improve the wearable technolo- 
gies for the best user experiences. A methodology to build this 
kind of wearable technology is introduced in Section VI-B. 
The information is stored in the Database Knowledge for user 
feedback purposes or for providing input in the Data Analysis 
System. 

B. Building Wearable-in-the-Loop 

This methodology provides a mechanism for creating wear- 
able appliances and integrating them with third part systems 
in order to get achieve maturity on the HITL process. A 
wearable appliance was built and evaluated for application in 
natural environments. The purpose of this section is present a 
methodology for building wearable appliances. 

Each interaction in this loop represent an opportunity to 
improve the user experience and wearable features. In the 
following, each stage of the methodology is explained: 

a) Define System Requirements: In this stage, it is impor- 
tant to consider the wearable requirements such as usability, 
weigh, size and battery consumption. 

b) Develop Application: Development of software and 
the design of the wearable system is performed. 

c) Evaluate Accuracy and Application Performance: 
Evaluation of the software accuracy is required to determine if 
the functional requirements are satisfied. This should be before 
the software is embedded into the hardware. 

d) Choose Hardware: The hardware, in accordance with 
the system requirements, should be defined for posterior 
evaluation. 

e) Embedded Application: Install the software in the 
hardware defined. 

f) Evaluate System Performance and Requirements: 
Define metrics and parameters to evaluate the hardware per- 
formance. It is also important to evaluate the wearable system 
requirements, such as battery consumption. 

g) Evaluate Human-Computer Interaction: Evaluate the 
usability of the wearable for humans performing their tasks. 

h) Apply the System in an Industrial Environment: Use 
the system in the target industrial scenario. 
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Fig. 1. Wearable-in-the-loop Methodology 


i) Send Data to Wearable-in-the-loop System: The data 
collected by the wearable system for workers should be stored 
in the system integrated to the WITL. 

j) Get New Mature Stage: If there is new knowledge use 
it to improve the wearable system and the user experiences. 

C. Wearable Example 

This section presents a wearable construction using the 
methodology presented in the previous section. The real-time 
detection of pupil center location provides valuable informa- 
tion to be utilized in a wide range of applications. Gaze 
tracking systems are a type of application which uses the pupil 
center location to infer the direction of the users gaze. 

To develop a new algorithm that represents the state-of-the- 
art of pupil center location identification is not the focus of 
this paper. The algorithm presented by [13] has been used 
and embedded on different boards to evaluate it performance. 
Next sections describe the wearable system according to 
the methodology proposed. We give an example which uses 
the proposed method up to the ’’Evaluate Human-Computer 
Interaction” stage, because we do not receive a license to 
implement this system in an industrial environment. 

System Requirements: The wearable system needs to iden- 
tify the user pupil center in real time, and provide this 
information at least twice per second, in case of low eye 
motion speed, or at least four times per second, in case of high 
eye motion speed [14]. Therefore, the hardware and software 
need to have a minimum frame rate between 2 and 4 FPS to 
provide this requirement. 


Hardware Selection: We choose four embedded platforms 
to assess the hardware and software performance. Each one 
is in according with the wearable requirements and can work 
with a battery as a power supply. The following embedded 
platforms were selected: Intel Edison (Dual-threaded Intel 
Atom CPU at 500 MHz, a 32-bit Intel Quark microcontroller 
at 100 MHz and 1 GB LPDDR3 RAM, 1MB cache LI), 
Wandboard Quad (Freescale i.MX6 Quad core processor 
at 1GHz and 2GB DDR3 RAM), Raspberry Pi 3 model b 
(Broadcom Quad-Core BCM2837 64 bits, 1.2GHz and 1GB 
SDRAM), Cubie board (ARM Cortex A8 at 1 GHz 1 GB 
DDR3). 

Application Development and Accuracy Evaluation: The 
algorithm presented in [13] was implemented using the Python 
language and OpenCV library. We choose this language due 
to the fact that all functions and methods proposed and used 
by [13] are introduced in this API. We did not present a study 
about accuracy evaluation because the authors present it in the 
previous paper. 

Embedded Application: For each development board, we 
use Linux as the operating system. For Intel Edison, we used 
the Yocto framework to build the operating systems. Each 
one uses the minimal core building, since our intention is 
to optimize the hardware performance. After the operating 
systems installation, we build the OpenCV library with Python 
dependencies to execute the algorithm. 

Evaluate System Performance and Requirements: Frame 
per second is the metric defined to evaluate this system. We 
use a video stream to evaluate the hardware performance. 
Although the system uses a camera as sensor, we use this 
approach in the first evaluation to avoid camera interference 
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Fig. 2. Hardware Performance 


on the algorithm performance. The equipment used executes 
at maximum 30FPS. 

Figure 2 shows the results obtained after the evaluation with 
99% of confidence. We execute the sequential and parallel 
versions of the algorithm. The parallel version divided the 
image into two parts (one region of interest for each eye) 
and each one is sent to a core of the board. All development 
boards had a satisfactory frame rate for this application. 

The frame rate difference between single-core and multi- 
core versions is not significant. Intel Edison has the biggest 
difference with two frame rate more in the multi-core version 
than single-core version. This is relevant information to be 
considered in the wearable-in-the-loop methodology because 
the next wearable version can demand process the execution 
of other processes. 

Prototype Construction and User Experience Evaluation: 
We use a safety helmet to build the prototype. This device is 
according to the industrial environment. We also avoid making 
modifications to the equipment. We choose the Intel Edison 
due to its size and weight. Other boards are too large to be 
adapted on the helmet without changing its structure. 

Figure 3a shows the camera and the Intel Edison assembled 
into the safety helmet. It was necessary create a base to fasten 
the camera because the field of view of the selected device 
demand about 6 cm to capture both eyes. We can use a 
different lens or another camera with a better field of view 
configuration to avoid this base and optimize the size of final 
prototype. Figure 3b shows a side view of the installed camera 
and a human using the equipment. Figure 3c shows an image 
captured by the prototype camera. The image contains both 
eyes of the user as necessary in the implemented algorithm. 
Finally, Figure 3d shows the user view. Note that a requirement 
of the application is not to block the user view. A small part 
of the camera is in the field of view of the user, however, 
the correction of the camera’s field of view configuration may 
avoid this part in front of the user. 



Fig. 3. Final Prototype 


VII. Conclusion 

In this paper, we have presented the use of wearable 
technology as an appliance for human-computer interaction 
in Industry 4.0 environments. A discussion about the human 
aspects and human interaction in CPS has been presented. 
The technological aspects for building wearable appliances 
were also discussed. Finally, a methodology for building 
wearable appliances has been presented based on our previous 
experience, together with an methodology to improve the user 
experience in the industrial scenario. We also have presented 
a wearable case study built with the proposed methodology. 

This paper presents and illustrates the first part of wearable 
in the loop. As future work, we will apply a case study 
in a industrial environment. Also, a study about the human 
influence on the cyber-physical systems will be done. We are 
planning to use the wearable system to control and configure 
the robots in the industrial environment. 
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